IV Regularization Tools for TrainingLarge - Scale Neural Networks

We present regularization tools for training small-and-medium as well as large-scale artiicial feedforward neural networks. The determination of the weights leads to very ill-conditioned nonlinear least squares problems and regularization is often suggested to get control over the network complexity, small variance error, and nice optimization problems. The algorithms proposed solve explicitly a sequence of Tikhonov regularized nonlinear least squares problems. For small-and-medium size problems the Gauss-Newton method is applied to the regularized problem that is much more well-conditioned than the original problem, and exhibits far better convergence properties than a Levenberg-Marquardt method. Numerical results presented also connrm that the proposed implementations are more reliable and eecient than the Levenberg-Marquardt method. For large-scale problems, methods using new special purpose automatic diierentiation combined with conjugate gradient methods are proposed. The algorithms developed utilize the structure of the problem in diierent ways and perform much better than Polak-Ribi ere based methods. All algorithms are tested using benchmark problems and guidelines by Lutz Prechelt in the Proben1 package. All software is programmed in Matlab and gathered in a toolbox.